Introduction

So far, we have dealt with situations where everyone who is assigned to treatment actually receives the treatment.

But this may not always hold:

Further, the people who get treated may be systematically different from the people who are assigned to the treatment group, but somehow don’t get treated.

What types of causal effects can we estimate in such situations?

A Motivating Example

Suppose that you are interested in assessing the effect of a face-to-face canvassing effort on voter turnout. Here’s the design:

Suppose you were the principal investigator of this study. How should the data from these three groups be analyzed? Which groups (boxes) would you compare?

Some Terminology

Importantly, these designations define how treatment status (D) responds to treatment assignment (Z). They are not defined with respect to potential outcomes (Y).

Calculating different effects with full information

Hypothetical Schedule of Potential Outcomes assuming One-Sided Non-Compliance
Y(d=0) Y(d=1) D(z=0) D(z=1) Type
A 4 6 No Yes Complier
B 2 8 No No Never-Taker
C 1 5 No Yes Complier
D 5 7 No Yes Complier
E 6 10 No Yes Complier
F 2 10 No No Never-Taker
G 6 9 No Yes Complier
H 2 5 No Yes Complier
I 5 9 No No Never-Taker

Questions:

  1. What is the “true” ATE? That is, what is the difference in Y if all subjects complied with their treatment assignment?
  2. What is the intent-to-treat effect (ITT)? That is, what is the difference in Y based on subjects’ actual treatment status?
  3. What is the ATE amongst compliers? Let’s call this the Complier Average Causal Effect (CAGE).

A graphical illustration

\[CACE = \frac{ITT}{Compliance Rate} = \frac{2}{0.67} = 3 \]

Some subtle points:

  1. The figure illustrates why the “true” ATE cannot be estimated from an experiment with non-compliance.
    • The empty box in Panel A represents the average treatment effect among Never-Takers.
    • If Never-Takers were (somehow) exposed to the treatment, the empty region and the cross-hatched region would together sum to the “true” ATE.
    • But the empty box never materializes because our experiment fails to treat Never-Takers.
  2. Increasing the compliance rate could raise or lower the CACE.
    • Increasing the compliance rate means increasing the denominator in the CACE equation (so we divide by a larger number)
    • But in our specific example, treating never-takers would also increase the numerator
  3. Relatedly, the “true” ATE is a weighted average of the ATE for compliers and the ATE for never-takers.
    • But, in the real world (unlike our hypothetical example here with full info), you have no idea whether these two quantities are the same.
    • So be careful about generalizing the CACE to make inferences about the ATE for the sample as a whole.

Estimation (when we don’t have full information)

Let’s return to our voter mobilization example. Imagine we get the following results:

  1. What is the ITT?
    • The ITT can be informative, as it summarizes the net impact of an intended intervention. In these cases, failure-to-treat may be a “feature” of how the intervention will work in the real world, and not a “bug”.
    • But often researchers also want to learn about the effectiveness of the treatment that was administered, not the treatment that was assigned. This information is provided by the CACE (but only for a subgroup of subjects).
  2. What is the compliance rate?
    • Note: technically, the compliance rate = (% treated if assigned to treatment) - (% treated if assigned to control)
    • Here, (% treated if assigned to control) = 0 by design
    • But the same cannot be said in an experiment with two-sided non-compliance where we must also account for always-takers
  3. What is the CACE?
  4. What happens if we compare the turnout rate among those actually contacted vs. the turnout rate in the control group?
    • Can you think of substantive reasons why this comparison would be upwardly biased compared to the CACE?
  5. What happens if you compare the turnout rate among those actually contacted vs. those never contact (in both T and C)?

Another problem to work through

Suppose you conducted a field experiment where canvassers visited homes and encouraged residents to recycle.

When implementing the intervention, you encountered one-sided noncompliance: 1015 of the 1849 homes assigned to the treatment group were successfully canvassed; none of the 1430 homes assigned to the control group were canvassed.

When measuring outcomes three weeks later, you find that:

You also observed that:

Questions:

  1. What is the ITT?
  2. What is the CACE?
  3. Explain why comparing the recycling rates of the treated and untreated subjects tends to produce misleading estimates.

Assumptions

Non-interference:

Exclusion restriction:

Here’s an example of a violation (e.g. if someone is not home when canvassers visit, canvassers leave a flyer encouraging people to vote):

Estimating using instrumental variables (IV) regression

Download a toy example of the recycling data here.

First stage: regress treatment status (D) on treatment assignment (Z)

\[D = \alpha_1 + Z + \epsilon_1\]

Second stage: regress outcomes (Y) on predicted treatment status (\(\hat{D}\)) from the first stage

\[Y = \alpha_2 + \hat{D} + \epsilon_2\]

You can do both simultaneously using 2sls (ivregress command in STATA).

With a small proportion of compliers,

\[SE(\hat{CACE}) = \frac{SE(\hat{ITT})}{ComplianceRate}\]